1. Field of the Invention
The present invention relates to digital signal analysis techniques to test interactive software application(s) running on a System Under Test (SUT).
2. Related Art
Traditionally, the testing of interactive software applications require that an operator enter input data into a system under test (SUT), using a keyboard or other input device. As the input data is entered, the operator is then responsible for observing the results on a visual display screen of the SUT to determine if the software application being tested produces the expected result. This type of testing of interactive software applications is often referred to by those skilled in the art as regression testing.
Regression testing serves important functions throughout the software development industry. For example, during the process of developing a software application, it is often desirable to run each new version of a software application through a regression testing procedure to ensure that it responds to the input data in the expected manner. In addition, the fact that a software application under test performs as expected can be used as an indication that the hardware on which the software application runs is also performing as expected.
As noted above, regression testing has traditionally been performed manually (that is, a human operator enters the input data for each test). Manual regression testing of interactive software applications, however, is inadequate in several ways. First, manual regression testing is costly, since it is both time and labor intensive. Second, manual regression testing is inefficient and error-prone, since the operator conducting the test may fail to observe any improper visual display screen images of the SUT or enter inappropriate input data. An improper video display screen indicates to the operator that the SUT has not interpreted the input data correctly or that the SUT hardware has failed in some way. In addition, manual regression testing does not allow for production testing or development purposes, since real-time performance measurements and compatibility comparisons of the software cannot be efficiently made.
Automated means for performing regression testing of interactive software applications is therefore considered desirable.
One of the first methods for automating such regression testing of interactive software applications involves fixed interval or pacing systems. Such systems connect to the SUT user device interface ports and provide the input data to the software application running on the SUT at fixed intervals of time.
Fixed pacing systems such as these, however, are undesirable for several reasons. Since no feedback is being gathered from the SUT, there is no indication that the SUT is executing the software application correctly. It is often the case that a fixed interval is impossible to predetermine, and overrun results. Also, since the input data is sent to the SUT at fixed intervals, performance measurements of the software applications and the hardware of the SUT cannot be measured. Moreover, without some type of feedback, it is impossible to determine if the SUT is actually displaying the expected images on its video display screen.
Several later efforts to achieve an automated solution to regression testing are inadequate in similar ways. Attempts at applying software patches or modifying actual software application code to create feedback internal to the code are inadequate because they cause changes in the behavior of software applications in very unpredictable ways. These behavioral changes further prohibit accurate performance measurements from being made of software applications.
Automan, jointly developed by Hewlett Packard Corporation of Palo Alto, Ca. and Telamon Corporation of Oakland, Ca., is a system designed for automated regression testing of interactive software applications. However, the Automan system was designed for testing interactive software applications running on a computer which utilizes a terminal. As such, Automan is based on the existence of a data stream passing over a communication line between the terminal and the computer running the software application using a communication protocol. Because Automan requires this data stream and terminal communication protocol, it is not directly applicable to personal computers, which integrate the keyboard, display and processor.
Moreover, Automan only monitors the data stream for expected characters. Automan does not provide a means for ensuring that the actual visual display screen image is correct. Thus, if the test has failed in some way, it is very possible that while correct data passed to the terminal, the visual display screen was nonetheless incorrect.
Another method of performing regression testing of interactive software applications involves monitoring the image of the visual display screen and periodically capturing the visual display data (which controls the image on the visual display screen) on the SUT's visual display device. Once the visual display data is captured, that is, intrusively accessed by an outside system, a comparison of this.. data is made against earlier received visual display data.
This method is undesirable because it is extremely invasive to the SUT, requires large blocks of memory for visual display data captures, and destroys any real-time nature of the tests.